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Abstract 

Background: A 'societal impact factor' that complements the scientific impact factor would contribute to a more 
comprehensive evaluation of scientific research. In order to develop a practical tool for its assessment, it is 
important to learn about perceptions of scientists on how to measure a societal impact factor. 

Methods: This qualitative study presents the development of a practical tool to measure the societal impact of 
publications based on 8 focus group discussions with 24 biomedical scientists at the Medical University Vienna 
between May 2008 and May 2009. Topics focused on (1) features of an ideal tool, (2) criteria that should be 
considered in the assessment, and (3) the identification of practical pitfalls. In an iterative exercise involving the 
repeated application of the drafted tool to scientific papers, criteria for the assessment were refined. A small-scale 
exercise to evaluate the tool in terms of its comprehensibility, relevance and practicability was conducted using 
questionnaires for 6 external experts in leading positions of public health, and yielded acceptable results. 

Results: The tool developed consists of three quantitative dimensions, that is (1) the aim of a publication, (2) the 
efforts of the authors to translate their research results, and, if translation was accomplished, (3) (a) the size of the 
area where translation was accomplished (regional, national or international), (b) its status (preliminary versus 
permanent) and (c) the target group of the translation (individuals, subgroup of population, total population). 

Conclusions: Focus group discussions with scientists suggested that the societal impact factor of a publication 
should consider the effect of the publication in a wide set of non-scientific areas, but also the motivation behind 
the publication, and efforts by the authors to translate their findings. The proposed tool provides some valuable 
insights for further research and practical applications in the topic area. 



Background 

To date, more than thousand different scales exist which 
aim at measuring scientific performance. The most com- 
monly used metrics are citation-based [1], with the 
Institute of Scientific Information (ISI) impact factor 
representing the most widespread measure. The impact 
factor has been shown to influence editorial decision 
making as well as author citation behaviour [1-4]. In a 
recent poll including 150 Nature readers, 50% of the 
respondents said that they had shaped their research 
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behaviours on the basis of the metrics used at their uni- 
versity [5,6]. There has been increasing criticism in the 
impact factor's use as the gold-standard of measuring 
scientific performance [7]. Specifically, the common 
practice of using the scientific impact factor for measur- 
ing performance of individual researchers has been 
repeatedly criticised for being inappropriate in the light 
of what the impact factor actually measures [1,7]. To 
this criticism, recent research on the relatedness of dif- 
ferent scientific impact measures adds that the impact 
factor does not seem to be situated at the core of the 
notion of scientific impact but represents a rather speci- 
fic aspect of scientific impact [7]. 

The ever-increasing demand for research that makes a 
practical difference in terms of translated outcomes 
reflected, among others, in policy and societal health, 
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has given rise to ideas about the necessity to evaluate 
scientific research not only in terms of the accomplished 
scientific impact, but also in terms of policy impacts, 
service impacts or societal impacts [8-16]. In particular, 
the societal impact has been repeatedly discussed as a 
relevant aspect to determine the value of a publication 
for the society [8,9,13,14,17]. 

Currently, research which focuses on the question 
how a societal impact of research may be practically 
assessed is sparse. One of the first approaches to assess 
wider impacts of medical research, with a specific 
emphasis on economic payback, is the Buxton and Han- 
ney Payback Framework [18]. In this framework, five 
main categories of payback are investigated, namely 
knowledge, research benefits, political and administrative 
benefits, health sector benefits, and broader economic 
benefits. A core feature of this model is an investigation 
of the interfaces between research and its environment. 
The model has been successfully adapted and applied to 
many different fields in recent years including but not 
limited to health economics, health technology, health 
services research and clinical research fields [18,19]. 
Another British project investigated several dimensions 
of impact of public policy projects, epidemiology and 
infectious diseases. It provides a framework which 
assists researchers in describing the effects of their work 
in different impact areas and includes considerations on 
the societal impact of research [20]. Even more specifi- 
cally related to the area of societal impact, a Dutch 
ongoing project has developed an outline for an evalua- 
tion methodology as a basis for further development. 
This outline includes potentially suitable indicators of 
the societal impact of research [10,11,17]. It was devel- 
oped based on the input from a wide range of disci- 
plines including health sciences, architecture, law, social 
and technical sciences as well as philosophy. Another 
Dutch project focused on the societal output at the level 
of medical research departments at the Leiden Univer- 
sity Medical Center, and defined stakeholder specific 
indicators of societal impact in the areas of knowledge 
production, knowledge exchange, knowledge use and 
earning capacity [21]. A further structured approach to 
the assessment of indicators that demonstrate research 
impact beyond citation count and targets primarily at 
basic medical and clinical research settings was provided 
by a US project [15]. 

Studies related to the perceptions and opinions of 
scientists on the societal impact of their research publi- 
cations are currently lacking. Because scientists in 
applied fields such as public health constitute a highly 
important group of stakeholders in the translation of 
research into practice, knowledge focusing on their opi- 
nions may contribute to the ongoing endeavours of 
defining the societal impact of scientific reasearch. 



Current literature shows that health scientists in applied 
academic settings are not satisfied with the ways of how 
the impact of their research (which normally means, the 
scientific impact of peer-reviewed publications) is evalu- 
ated [1,2,4,6,8,9,22]. This dissatisfaction is partially due 
to the lack of acknowledgement of the societal effects of 
research in the assessment [9,13,14,17,22]. It was, there- 
fore, the aim of a much-noticed process launched at the 
Medical University of Vienna [8,22] to qualitatively 
assess perceptions of biomedical scientists in the public 
health area about how to assess the societal impact of 
publications and to create a practical tool to calculate a 
societal impact factor. 

Methods 

Focus group discussions as a method for investigating 
group opinions 

Focus group discussions were used as the primary 
method for investigating opinions on the societal impact 
of research publications in the present study. This 
approach is useful for obtaining information about 
knowledge, ideas, opinions and attitudes in a community 
or group, particularly when little preknowledge about an 
issue is available [23,24]. As compared to individual 
interviews, group discussions enhance creativity because 
they allow for interaction among group members. They 
are therefore appropriate for the identification of pro- 
blem-based solutions [24]. The interactive component 
has been discussed to "empower" research participants 
because they allow participants to become an active part 
in the process of developing ideas [24]. Focus group dis- 
cussions, however, also have important downsides. The 
interaction of group members which may on the one 
hand enhance creativity, may on the other hand silent 
individual voices of dissent [24]. Therefore, a well-con- 
sidered approach is necessary when composing groups, 
conducting the interviews and interpreting results 
[23,24]. 

Formation of the Societal Impact Factor Task Force 

The Centre for Public Health comprises 5 departments 
dedicated to general practice and family medicine, 
social medicine, environmental health, epidemiology 
and medical psychology, respectively. An invitation to 
join a task force for the development of a tool to 
assess the societal impact of publications based on 
focus group discussions was sent by email to all scien- 
tific employees of the Centre for Public Health, Medi- 
cal University Vienna in December 2007 by the 
moderator Manfred Maier (MM). Twenty-four scienti- 
fic employees, with representatives from all depart- 
ments, accepted the invitation and formed part of the 
task force (for names and affiliations, see acknowledge- 
ment section). 
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Mixed purposeful sampling strategy 

The outlined composition of the Centre is comparable 
to many other international academic public health 
institutions, which makes the Centre an appropriate 
place to study opinions among a purposeful typical case 
sample of scientists in an academic public health setting 
[25]. In order to purposefully eliminate highly influential 
opinions, which may disproportionally influence the 
content of the discussions, the opinions of the depart- 
ment chairmen were analyzed in an opinion leader 
group discussion separate from the other participants 
[23]. The other groups each involved 5 to 7 scientific 
employees from different departments. Researchers from 
different departments were mixed to ensure a more het- 
erogeneous sampling of individuals within the relatively 
homogenous setting of the Centre, which is expected to 
result in more diverse opinions and the breaking up of 
hierarchical structures which may influence the contents 
of the discussions [23,24]. 

Conduction of the focus group discussions 

Two rounds of 8 focus group discussions in total, each 
involving 5 to 7 members of the task force, were con- 
ducted between May 2008 and May 2009. 

The first round of focus group discussions was dedi- 
cated to the questions of (1) what an ideal tool to mea- 
sure the societal impact should be like, and (2) what 
specific indicators should be considered when quantita- 
tively assessing a societal impact of publications. The 
focus groups typically started with an outline of the gen- 
eral purpose of the discussion, and some general warm 
up questions, followed by a clarification of terms used 
in the discussion when necessary. The moderator (MM) 
encouraged all participants to actively participate in the 
discussion. The discussions involved individual state- 
ments by the participants and group interaction. 

The contents of the discussions were recorded in writ- 
ing by an assistant and/or the moderator. 

The qualitative data for each group discussion was 
analyzed inductively with qualitative content analysis, 
and codes were assigned to the themes related to the 
explored questions occurring in the discussions. Themes 
that were mentioned repeatedly were then used to con- 
struct a preliminary tool, which was a questionnaire for 
the self-assessment by authors of the societal impact of 
own publications. To increase precision, the moderator 
and 2 other task force members (TN, TED) collaborated 
on the transformation of the qualitative codes into the 
self-assessment form. The self assessment form was 
then sent out to the Task Force members who were 
asked (1) to apply the form to one own selected publica- 
tion and (2) to review the self-assessments of two or 
three colleagues. The members were asked to provide 
feedback on (1) their perceptions on the suitability of 



the tool, (2) practical experiences in the application of 
the tool to his or her own publications, and (3) practical 
experiences in the assessment of publications from 
other authors in a further round of focus group meet- 
ings, following the same approach as in the first round. 
This feedback was then used for a revision of the self- 
assessment questionnaire. A code book defining the 
terms used for the assessment and the variables (codes) 
used to assess the societal impact was developed based 
on the contents of the focus group meetings, in addition 
to an information sheet for reviewers and authors which 
aimed at supporting authors and reviewers in their tasks 
within the application process. 

Exercise to evaluate the tool 

To perform a first exercise related to the validation of 
the tool developed, six international external experts 
with leading roles in academic public health were 
invited to test the tool and to provide structured com- 
ments. TN and TED developed a questionnaire which 
allowed the experts to rate the comprehensibility, the 
relevance and the practicability of the variables used in 
the societal impact factor assessment questionnaire, and 
the comprehensibility and usefulness of the glossary and 
of the information to authors and reviewers on 4-degree 
Likert scales (1: 'comprehensive", 'very relevant", 'very 
practicable', 'very appropriate', 'very useful', and 4: 'not 
comprehensive at all', 'not relevant at all", "not practical 
at all', 'not appropriate at all', 'not useful at all'). When- 
ever 3 or 4 was ticked, the experts were asked to specify 
the reasons of their choice. In addition, the question- 
naire included open-ended comments related to the 
tool, e.g. the time needed for the assessment. 

This questionnaire, together with three representative 
publications from different research areas at the Centre 
for Public Health, the respective self-assessment forms 
by authors, a short version of the code book to be used 
as a glossary, and an information sheet for authors and 
reviewers as a support material was sent to 6 external 
experts. Publication A used for this test was a national 
health report on diabetes mellitus type 2 in Austria [26]. 
In this report epidemiological data on diabetes, risk fac- 
tors and complications were compiled and analysed, and 
national prevention strategies and policy implications 
were made [27]. Publication B dealt with mental health 
promotion and behavioural medicine. It analysed the 
impact of media guidelines for reporting on suicides on 
the quality of suicide reporting and on suicide rates in 
Austria [28]. Publication C was a survey on the alloca- 
tion of training posts to applicants for postgraduate 
medical education in Austria [29]. The experts were 
selected based on their leading academic roles at depart- 
ments of Public Health or General Practice (for names 
and affiliations, see acknowledgement section). 
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Ethics statement 

The study was conducted in compliance with the Hel- 
sinki Declaration. All participants in the focus group 
discussions are investigators and co-author the paper 
and the drafted tool. The need to obtain ethical consent 
for this study was specifically waived by the Institutional 
Review Board at the Medical University of Vienna, 
Austria. 

Results 

Focus group discussions 

Ideal features of a tool to measure the societal impact 

A frequently mentioned feature of an ideal tool to 
measure the societal impact of a publication was good 
practicability in the sense of quick and easy to use 
application. One participant argued that easy applica- 
tion would be crucial for successful implementation in 
an academic setting. Related to this theme, partici- 
pants also mentioned that only the most relevant 
aspects should be used for assessment. Another fea- 
ture which appeared frequently was that the tool 
should be comprehensive enough not to leave out 
important aspects of a societal impact. There was a 
main consensus in the focus groups that relevant 
aspects may be different for different public health 
related fields, and that this implies that an appropriate 
tool should leave room for a wide set of field-specific 
relevant aspects. 

Indicators to be considered when quantitatively assessing a 
societal impact of publications 

The themes that emerged here were situated either at 
the level of the effect of the publication, or at the level 
of its original aim. Furthermore, intellectual and practi- 
cal investments by the authors were considered impor- 
tant by several participants. Related to the effect level, a 
main point raised in all discussions was how to appro- 
priately show the relatedness of a specific translational 
outcome to a publication. Several participants argued 
that it should be the task of the authors to outline a 
causal chain between the specific publication and a spe- 
cific outcome. 

There were a multitude of views about which effects 
should be counted as a societal impact, which resulted 
in a broad consensus that an appropriate tool should 
allow for an individual description of the specific effect 
related to the publication achieved. In all focus groups, 
time and space issues related to the accomplished effects 
were raised. Several participants felt that lasting transla- 
tions or translations that go beyond the regional level 
should be considered of greater societal impact than 
smaller translations. Several participants also mentioned 
that the number of individuals influenced by the accom- 
plishment should somehow be considered in the 
assessment. 



At the level of the investments by the authors, several 
participants mentioned that most, if not all translation 
effects accomplished required some efforts by the 
authors. However, there were also a few participants 
who argued that personal efforts would not be necessary 
for a translation of research results. Many different 
views were present for the type of effort that should be 
counted as a means to achieve translation. Some partici- 
pants argued that basically all activities beyond the pub- 
lication should be counted as translational effort, others 
argued that the societal impact should complement the 
scientific impact and should focus more on aspects not 
normally represented in scientific work. 

Related to the level of the aim of the publication, 
there was no clear consensus about its role for the 
assessment of the societal impact. Several participants 
discussed that not only the act of 'producing new 
knowledge' may result in a societal impact. Also aims 
such as the increase of awareness among the readership 
or the application of existing evidence for a practical 
purpose was argued to be an aim relevant for translation 
later on. 

Tool to assess a societal impact factor 

Based on the findings in the focus group meetings, the 
questionnaire developed consisted of three quantitative 
dimensions of assessment, that is (1) the aim of a publi- 
cation ((a) gain of knowledge, (b) application of knowl- 
edge, and (c) increase in awareness), (2) the specific 
efforts of the authors to translate their research results 
into societal action, and, if translation was accomplished, 
the size of the translation in terms of (3) (a) the size of 
the geographical area where the translation was accom- 
plished (regional, national, international), (b) its status 
(preliminary versus permanent) and (c) the target group 
of the translation (individuals, subgroup of the popula- 
tion, total population). For reasons of simplicity, MM, 
TN and TD agreed to assign one societal impact factor 
point to publications which are consistent with any of 
the outlined aims, one additional point for any efforts 
taken by the authors; one, two, and three points, respec- 
tively, for a regional, national and international transla- 
tion; one and two points respectively, for a preliminary 
and permanent translation, and one, two, and three 
points, respectively, for translations targeting individuals, 
subgroups of the population or the whole public. It was 
agreed upon that, at a later point in time, the quantitative 
categorization would have to be adapted, based, among 
others, on the distribution of points achieved by a larger 
amount of studies assessed for a societal impact. 

The final versions of questionnaire used as the (self- 
assessment) application form and the assessment form for 
the reviewer, are given in Figure 1 and additional file 1, 
respectively. The final version of the glossary is given in 
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SOCIETAL 


IMPACT 


APPLICATION FORM FOR AUTHORS 


NAME OF AUIHOR(S): 




TITLE OF PUBLICATION AND SOURCE: 

First application: yes no (last score for societal impact: ) 
Otherrelevantpublications: enclose list 


AIM OF PUBLICATION 


AUTHOR'S 
oLLr -t \ AL LAI 1U.> 


AUTHOR'S JUSTIFICATION'S AND/OR EXPLANATIONS (please enlarge space if necessary): 


i t .h i 'i o f kn owl ed ge OR 
application of knowledge 
OR increase in awareness 


□ yes (1) 

□ no (0) 




TRANSLATION 






Specific activities or 
initiatives undertaken 


□ ves (1) 

□ no (0) 




accomplished 


□ yes 

□ no 




LF TRAN SLATION ACC. 






Level 


□ regional (1) 
l_l national (2) 

□ international (3) 




Status 


□ preliminary (1) 

□ permanent (2) 




Target group 


D individuals (1) 

□ subpopulations (2) 

□ public (3) 




Total score 







1 declare that the publicationsubmitted is original and had an impact on health, society oj practice reality as describedaboxeandvas not primarily a means to support translation and 
implementation of researchresults publishedprexiously in another journal by me the authors. 

Author s signature. at. in. 



Figure 1 Application form for a societal impact factor to be used by the applicant for self-assessment 



table 1. The information to authors and the information to 
reviewers are shown in Figures 2 and 3, respectively. 

Expert evaluation 

The comprehensibility, relevance and practicability of 
the items used for the assessment of the societal 



impact factor, the glossary as well as the information 
to reviewers and to authors were rated as acceptable 
(1 or 2) by all external reviewers. The only excep- 
tions were the relevance and the practicability of the 
item 'target group", which were rated as 3 by one 
reviewer. The appropriateness of the complete set of 



Table 1 Glossary provided to authors and reviewers 



Societal impact 



Societal impact factor 
Publication 

Publication aim 
Translation 

Accomplishment (of 
translation) 



The use of a research publication in a non-scientific area relevant to the society (e.g. in mass media reports, policies, 
guidelines). 

Measure to quantify the societal impact. 

Scientific publications in journals with or without scientific impact factor and research-based publications in other 
media. 

Aim to increase knowledge, to apply knowledge and/or to increase awareness. 

Any efforts to make research results applicable to the population (e.g., press conference, mass media report, educative 
event). 

An end-point, or more frequently a milestone, at which activities to translate research results receive physical uptake/ 
implementation in an intervention, treatment, policy, or other health-related non-scientific application. 

When assessing the accomplishment of translation of research results into practice reality, it is relevant whether 

• the implementation was realized at the regional, national or international level, 

• the implementation reached a preliminary/pilot or permanent status 

• the implementation targeted at the individual patient case level, at a specified group of the population, or at 
the total population/society 
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Information for the author: 



The author has to complete the application form and has to provide information as follows with 
accompanying documentation in his or her application for a societal impact factor of his / her 
publication. 



1 . Publication aim: 

The author should explain if the aim of the publication was to gain knowledge, to apply knowledge or 
to increase awareness about the specific topic of the publication and provide evidence for his / her 
arguments wherever feasible. 

2. Efforts by the authors: 

The author should list all activities/initiatives ("efforts") she/he has undertaken to support translation 
of his/her research results into non-scientific areas / practice reality. He / She must attach documents 
supporting the arguments where appropriate. Examples are: press conference materials referencing 
the study, official information folder, press reports, interviews in newspaper/radio/television, reports 
in popular media, and papers resulting from conferences with politicians, opinion leaders, etc. 
Presentations at scientific meetings do not qualify as efforts in this sense. 

3. Translation accomplished 

If there was any translation of research results in the publication into non-scientific areas, the author 
(applicant) is asked to provide clear documentation of a causal link between the publication and the 
translation. This evidence is needed to support the self-assessment related to: 

• the "level of translation accomplished^ - either at the regional, national or international level 
(e.g., documented change in federal versus regional legislation which was influenced by the 
paper, documented uptake of research results into WHO or national guidelines, etc.). 

• the status of the translation - either preliminary (as indicated , e.g., by blue prints of WHO 
guidelines), or permanent (final product). 

• the target of the translation in terms of population size - either the individual patient level 
(individual cases but no specified subgroup of the population), subgroups of the population 
(e.g., groups of patients with a specific disease, elderly patients) or the level of the total 
population/public (e.g., many vaccination and general lifestyle issues, prevention of frequent 
chronic diseases). 

Figure 2 Information for authors 



items was rated as being acceptable (1 or 2) by all minutes and a median maximum time of 40 minutes 

reviewers. across reviewers. 

The time that the reviewers needed for the assessment The rating of publication A by the international 

of the societal impact of one publication ranged from 15 experts showed a median of 9 points (ranging from 7 to 

to 180 minutes, with a median minimum time of 20 10 points). Publication B showed a median of 9.5 points 
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Information for reviewers: 



The reviewer has to carefully consider whether the publication fulfills the inclusion criteria stated 
above or whether the respective publication is rather the means to support translation and 
implementation of research results published previously in another journal by the author(s). He /She 
will carefully evaluate the application filled in by the author and his/her arguments and documents for 
claiming points in each category of the assessment form. In order to receive a societal impact factor, 
the author has to achieve points in dimension 1 (publication aim) and additionally in dimension 2 
(specific efforts undertaken) and / or dimension 3 (translation accomplished). 

Figure 3 Information for reviewers 



(ranging from 8 to 10 points) and publication C a med- 
ian of 2 points (ranging from 2 to 6 points), as com- 
pared to 10, 10 and 2 points in the self-assessment 
(application) from the authors, respectively. 

Discussion 

The present project examined perceptions of scientists 
in different fields related to public health about the 
requirements of a tool to measure the societal impact of 
publications. Based on the insight gained in the focus 
groups and based on the repeated applications of the 
tool developed to scientific studies, it proposes a practi- 
cal tool to measure the societal impact factor of publica- 
tions. The tool comprises an assessment of the societal 
impact of a publication at 3 dimensions: (1) the aim of 
the publication, (2) the efforts of the authors to translate 
their research results into societal action or practical 
reality, and (3) the accomplishment of the translation in 
terms of the geographical reach (level), status (prelimin- 
ary or permanent) and size of the target group of this 
translation. A first exercise to evaluate the comprehensi- 
bility, relevance and practicability of the tool, which 
have been perceived as crucial features of a suitable tool 
in the focus groups, yielded encouraging results. The 
time needed by the reviewers for the assessment of the 
societal impact varied considerably between the experts 
and the different publications. 

The tool developed provides a novel complementary 
view to earlier projects by exploring the perceptions of 
scientists working in the field of public health. Com- 
pared to previously proposed assessment models, the 
results of the focus group discussions yielded an assess- 
ment model that was largely research-output oriented 
and is consequently suitable for post-hoc analysis of 
translation outcomes but not for prospective assess- 
ments of possible future translations of research, which 
are targeted at in other proposed assessment models 



[17]. In most cases, the societal impact of a publication 
is reached after the publication of the related study, and 
develops with time, often depending on the efforts of 
the researchers to translate their findings into societal 
action. The societal impact factor assignable to a publi- 
cation may consequently increase over time, e.g. due to 
a transition of the translation accomplished from a tem- 
porary to a permanent status, or due to an expansion of 
the translation to the international level following a 
translation at the regional level. To take future develop- 
ments into account, authors who apply for a societal 
impact factor should explicitly be invited to submit their 
publications for reassessment in case that there is some 
progress in the translation status. 

One of the main differences between the current and 
earlier approaches is that the proposed tool does not 
specify which kinds of translations (e.g., which resulting 
products, e.g. technical devices, guidelines, patents, etc) 
are specifically considered as accomplished translation. 
For example, a recently published model differentiates 
between biological materials, databases, patents, phar- 
maceutical products, curriculum guidelines, medical 
devices and many more, which all can be considered 
some kind of translation outcome [15]. In the present 
tool, it is completely left to the applicant to describe 
what his or her translation accomplishment actually is- 
the only prerequisites made are that there must be some 
documentation of a causal link between publication and 
the described beneficial translation outcome, and that 
the translation is accomplished in a non-scientific area 
relevant to the society. Compared to other definitions of 
societal impact used in the literature, the definition used 
here is wider than in other impact assessment models. 
This approach makes the tool easier to handle and 
applicable to different situations, research areas and 
practice fields. The reasoning behind the exclusion of 
impact in other scientific areas (as for example indicated 
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by the number of citations of the paper in scientific 
journals) from the societal impact assessment in this 
model was based on earlier suggestions that scientific 
citation-based assessments and societal impact assess- 
ments should be synergistic and complementary [21]. 

Several adaptations and specifications to the developed 
instrument were made based on the feedback from the 
external experts. Importantly, the aim of the publication 
was deemed to be a necessary but - alone - not a suffi- 
cient criterion for achieving a societal impact. Additional 
points in either dimension 2 (efforts undertaken) or 
dimension 3 (accomplished translation) were deemed 
necessary to achieve a societal impact factor. A further 
important specification was made for the 'status of the 
translation' in dimension 3. Here, it was deemed most 
suitable not to define a minimum duration of 'perma- 
nence' in the light of the ever-changing state-of-the-art 
practices in biomedical fields. Rather, a translation 
should be considered permanent if it is part of the offi- 
cial state of the art practice in its subject area at a speci- 
fic point in time. 

The third aspect in dimension (3), that is, (c) 'target 
group' (1 point: individuals, 2 points: subpopulation, and 
3 points: the public) required further specification of the 
terms used for the categories subpopulation and indivi- 
duals. In the present form of the instrument, the cate- 
gory 'subpopulation" should be used for any translations 
targeting directly a definable group of people (e.g., 
migrants; mentally ill people, depressed patients), 
whereas the category 'individuals" is for translations 
affecting individual cases of patients which cannot be 
described as a coherent group. The latter category may, 
e.g., be appropriate from the translation of a published 
case study. 

Strengths and limitations 

A strength of the present research is its focus on the 
perceptions of scientists related to the societal impact of 
health-related publications, with subsequent translation 
of the gained insights into a practical tool to assess the 
societal impact of publications. The focus group inter- 
views provided a relatively quick method to explore opi- 
nions in an interactive context and to develop creative 
problem-based solutions [24]. The repeated applications 
of the drafted tool to scientific publications and the 
expert feedback resulted in the identification of several 
practical issues and subsequent adaptations and refine- 
ments that are useful for further research activities and 
practical applications. 

The present study also has several limitations. The 
applied methodology of using focus group meetings has 
been criticised in the literature for not taking account of 
hierarchical structures in groups which influence the 
contents produced by the groups [30]. In spite of the 



fact that opinion leaders who may have a high influence 
on the other participants were purposefully sampled 
into one group and the moderator (MM) encouraged all 
participants to take equal part in the process, some 
related bias in the outcome cannot be ruled out. 

Furthermore, the tool developed can be used as an 
informative practical exercise, but it cannot be consid- 
ered to be ready for routine implementation. Adapta- 
tions and changes will be necessary before it can be 
used in various applied research fields, and it may turn 
out to be inadequate for specific fields that were not 
considered in the present study. Also, the quantitative 
point assignments to the different categories of the 
included items are preliminary at the present stage. A 
categorization into "low impact", "medium impact" and 
"high impact" is planned for the future when more 
assessments based on the tool become available. A 
quantitative assessment seems required to be able to 
implement the tool in academic settings that are cur- 
rently (nearly) exclusively considering quick and easy- 
to-use quantitative assessments of research outputs 
[5,8]. 

Implications for future research 

Focus group discussions seemed particularly suitable for 
the present study based on the small amount of pre- 
knowledge on the opinions and beliefs of researchers on 
the societal impact of their research publications [24]. 
Future work may use the ideas generated in this analysis 
to conduct individual interviews or to use consensus- 
building methods such as Delphi approaches, which can 
overcome some of the shortcomings of group discus- 
sions [31]. The analyses may be expanded to scientists 
from other fields than public health sciences. Addition- 
ally, other stakeholders who contribute to the formation 
of the societal impact may be included as participants. 
The tool developed in the present study acknowledges 
the contribution of other groups of stakeholders than 
scientists only indirectly by considering interdiscipinary 
(often non-scientific) work such as media collaborations 
and policy development. 

Conclusions 

Based on the views of scientists working in different 
fields of public health, the present study provides a 
novel exercise in developing a tool to measure the socie- 
tal impact of research publications. The findings provide 
ample materials to inform internationally ongoing pro- 
jects aiming at the development of measures to assess 
the societal impact of research publications. 

We encourage internationally ongoing research and 
application of the tool developed and ask colleagues to 
comment on the instrument at http://www.societalim- 
pact.info for improvements. 
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Additional material 



Additional file 1: Assessment form for a societal impact factor The 

assessment form for a societal impact factor which is to be filled out by 
the reviewer. 
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